Auditory}visual integration of talker gender in vowel perception

The experiments reported here used auditory}visual mismatches to compare three approaches to speaker normalization in speech perception: radical invariance, vocal tract normalization, and talker normalization. In contrast to the "rst two, the talker normalization theory assumes that listeners' subjective, abstract impressions of talkers play a role in speech perception. Experiment 1 found that the gender of a visually presented face a!ects the location of the phoneme boundary between [υ] and [S] in the perceptual identi"cation of a continuum of auditory}visual stimuli ranging from hood to hud. This e!ect was found for both &&stereotypical'' and &&non-stereotypical''male and female voices. The experiment also found that voice stereotypicality had an e!ect on the phoneme boundary. The di!erence between male and female talkers was greater when the talkers were rated by listeners as &&stereotypical''. Interestingly, for the two female talkers in this experiment, rated stereotypicality was correlated with voice breathiness rather than vowel fundamental frequency. Experiment 2 replicated and extended experiment 1 and tested whether the visual stimuli in experiment 1 were being perceptually integrated with the acoustic stimuli. In addition to the e!ects found in experiment 1, there was a boundary e!ect for the visually presented word: listeners responded hood more frequently when the acoustic stimulus was paired with a movie clip of a talker saying hood. Experiment 3 tested the abstractness of the talker information used in speech perception. Rather than seeing movie clips of male and female talkers, listeners were instructed to imagine a male or female talker while performing an audio-only identi"cation task with a gender-ambiguous hood-hud continuum. The phoneme boundary di!ered as a function of the imagined gender of the talker. The results from these experiments suggest that listeners integrate abstract gender information with phonetic information in speech perception. This conclusion supports the talker normalization theory of perceptual speaker normalization. ( 1999 Academic Press